Towards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks

Despite the recent success of deep neural networks in various applications, designing and training deep neural networks is still among the greatest challenges in the field. In this work, we address the challenge of designing and training feedforward Multilayer Perceptrons (MLPs) from a smooth optimisation perspective. By characterising the critical point conditions of an MLP based loss function, we identify conditions to eliminate local optima of the corresponding cost function. By studying the Hessian structure of the cost function at the global minima, we develop an approximate Newton’s MLP algorithm. Our results are demonstrated on an analysis of MLPs with only one hidden layer, and numerically evaluated on the benchmark problem of four region classification. Index Terms Forward Multilayer Perceptrons (MLPs), smooth optimisation, critical point analysis, Hessian matrix, approximate Newton’s method.